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This paper explores the concept of pragmatic meta-analytic studies in eLearning. Much 
educational technology literature focuses on developers and teachers describing and 
reflecting on their experiences. Few connections are made between these experiential 
‘stories’. The data set is fragmented and offers few generalisable lessons. The field 
needs guidelines about what can be learnt from such single-case reports. The pragmatic 
meta-analytic studies described in this paper have two common aspects: (1) the cases 
are related in some way, and (2) the data are authentic, that is, the evaluations have 
followed a naturalistic approach. We suggest that examining a number of such cases is 
best done by a mixed-methods approach with an emphasis on qualitative strategies. In 
the paper, we overview 63 eLearning cases. Three main meta-analytic strategies were 
used: (1) meta-analysis of the perception of usefulness across all cases, (2) meta- 
analysis of recorded benefits and challenges across all cases, and (3) meta-analysis of 
smaller groups of cases where the learning design and/or use of technology are similar. 

This study indicated that in Hong Kong the basic and non-interactive eLearning 
strategies are often valued by students, while their perceptions of interactive strategies 
that are potentially more beneficial fluctuate. One possible explanation relates to the 
level of risk that teachers and students are willing to take in venturing into more 
innovative teaching and learning strategies. 
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Summarising lessons learnt in naturalistic studies 

There are difficulties with the so-called ‘scientific research paradigm’ for education 
research. These include the complexity of interrelated factors in most educational settings; 
results from highly controlled scientific research designs not being replicable in real settings 
(Kember 2003); the difficulty of achieving matched groups in education; the ‘diffusion 
effect’ in controlled experimental designs in the same class or even institution whereby 
‘control groups can be contaminated’ (Craven et al. 2001, 643) by the experience aspects of 
the treatment; and the well known ‘halo effect’ (Thorndike 1920) whereby students who 
know they are part of a treatment group perform better just because they think they are 
getting special treatment. 

The naturalistic research paradigm thus seems better suited to educational research 
(Alexander and Hedberg 1994; Grasha 1990). Certainly, there is a large body of educational 
technology literature that focuses on developers and teachers describing and reflecting on 
their experiences with eLearning (Beetham 2005). Innovations are implemented in authentic 
educational settings and the collection of evaluation data is done in non-intrusive ways with 
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as little influence on the teaching and learning environment as possible. Naturalistic evalu- 
ation considers multiple perspectives and accepts that interpretation of interactions is not 
neutral and needs to be situated in a particular context of time and place (Froehlich 1994). 

Naturalistic research studies typically involve a mixed-methods approach with an 
emphasis on qualitative strategies, including observation, interviews, focus groups, 
questionnaires, narratives, and video- and audio-tape recordings. Qualitative descriptions 
‘provide a depth to student behavior that is largely absent from traditional measurement 
techniques’ (Grasha 1990, 108). In the same vein, Runciman (2002, 146) noted that ‘uncon- 
strained by the need to reduce the data to a set of numbers, qualitative research may allow 
phenomena to be studied from more perspectives and in greater depth’. Hammersley (1997) 
noted that an overemphasis on quantitative methods is not justified in educational research. 
Similar sentiments are echoed by Oliver and Conole (2003) who advocated the use of multi- 
ple models in eLearning evaluation and research in order to avoid an overly narrow view on 
what constitutes evidence. 

Secondly, multiple sources of data are often collected in a naturalistic study because 
obtaining accurate representations with qualitative data is often challenging. Qualitative 
data are often generated by small groups of students whose views may not represent the 
broader population (Hughes and DuMont 1993). Opinions collected and interpreted can be 
easily influenced by the researchers’ practice (Hague 1993). Triangulation of results 
through using multiple evaluation strategies is thus often used to build robust and more 
general interpretations of phenomena (Guba 1981). 

Thirdly, naturalistic evaluation studies may differ considerably. Different contexts and 
objectives for using a particular innovation usually result in a different set of evaluation 
strategies and instruments. The use of multiple sources of qualitative data makes comparing 
such studies even more difficult. 


Conceptual framework 

Three terms are central to this paper - ‘evaluation’, ‘pragmatic’ and ‘meta-analysis’. In 
educational evaluation, we propose that the intention is to gather evidence about the effec- 
tiveness of some educational practice or invention. The criteria for ‘effectiveness’ can vary 
markedly - evidence of ‘an increase in convenience’ is likely to be different from evidence 
of ‘an enhancement of student learning’. This is somewhat different from theory-oriented 
research where the intention is to develop or confirm specific theoretical constructs or 
models. While there is a clear overlap between evaluation studies and more theory-oriented 
research, the potential variety in design of evaluation studies brings a ‘fuzziness’ to the field 
and results in the need for guidelines about what can be learnt from single-case reports and 
which led to use of the term ‘meta-analysis’. We sought analytic strategies that could allow 
us to look across the data from a number of existing studies to reveal new patterns. 

In this study, our objective was to use the experience that already exists to find sugges- 
tions for practice that might be, to some extent, generalisable to other contexts, even though 
generalisability is not always the aim of evaluation and, indeed, is rarely so in single case 
studies. 

The study was ‘pragmatic’ in the sense that we looked for flexible principles that could 
guide our interpretations of existing data sets. 

The analysis focused on naturalistic studies because its aim is to inform practice. Qual- 
itative data play an important role because of the rich information they contain. We 
propose that generalisability of findings across a number of individual studies is enhanced 
using the following logic: 
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• If a phenomenon is found in several studies, then the phenomenon is more likely to 
be a general one. 

• If a simple general trend is not found, the meta-analysis across multiple cases may 
lead to the identification of some critical factors which can differentially affect 
success or failure in different contexts. 


Methodology 

Our meta-analytic study involved small-scale studies undertaken during the e3Leaming 
(enrich, extend, evaluate learning; e3L) project, which assisted teachers in using web- 
assisted teaching and learning in Hong Kong universities. It offered a range of design, 
development and evaluation services, and conducted a wide range of single-case eLearning 
evaluations. For each subproject or case, a team was formed whereby teachers had access 
to educational and technical support. Full details of the design of this project are in James 
et al. (2003) and the project website (http://e31earning.edc.polyu.edu.hlc). This paper 
develops the preliminary analysis in McNaught and Lam (2005a), involving the full e3L 
data set and introducing two additional meta-analytic methods to build our understanding 
of the issues operating to affect eLearning in Hong Kong universities. 

A tailor-made evaluation design was adopted for each case in order to serve the best 
interests of the individual teachers and cases (Lam and McNaught 2004). The customised 
approach resulted in highly individualised evaluation plans, questions and instruments, 
contrasting sharply with generic evaluation methods where standardised evaluation instru- 
ments are used across a range of sites. The process of evaluation consists of several 
stages: 

• working out the evaluation questions with the teacher; 

• choosing appropriate strategies to use in answering these questions; 

• developing an evaluation plan; 

• collecting data using the strategies selected (likely strategies were surveys, focus- 
group meetings, expert review and usability sessions), analysing the data; 

• reporting the evaluation results to teachers; and 

• assisting them in making decisions about what to do next. 

This mechanism provided useful feedback that informed not only the teachers involved 
in each subproject but also other teachers interested in using the strategies (Lam and 
McNaught 2008). Over a 30-month period, the e3L project supported the web development, 
implementation and evaluation of 70 subprojects designed to actively assist teaching and 
learning in courses. Sixty-three of these successfully reached the evaluation stage. The cases 
were very different in nature; Table 1 shows the range of web-assisted functions involved. 
Obviously, each of these cases involved usually more than one web-assisted function. We 
have described the functions in two ways. 

The first is the nature of the interactivity involved. Broadly speaking, interactivity can 
be either between the learner and online materials and tasks, or between people using the 
communication functions. Column 1 shows the four classifications we used: 

• low interactivity, basically accessing static information and information basic to the 
course (Low int.); 

• students interacting with materials and tasks (Ss and ‘stuff); 

• interactions about specific information (Info int.); and 


Table 1 . The nature of the web-assisted functions used in e 3 L cases. 
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• more public interaction space. In our e3L cases, the forums were used for: concept 
clarification, peer review, working on set cases, doing team assignments/projects, 
debates, socialising/building community, special topics/issues/news, and self- 
reflection (Public). 

We also classified the web-assisted functions in terms of four broad areas of web 
functions: 

• study management and skills support (Support); 

• resources/content for learning (Content); 

• assessment and feedback to students (Feedback); and 

• communication between teacher and students (Discussion). 

These classifications allowed the classification of eLearning strategies followed in the 
subprojects. Seventeen such strategies were identified, based on both the technology use 
and learning design. For example, ‘assignment submission’ and ‘exhibition of assignments’ 
were differentiated since the former are primarily assignment dropboxes while the latter 
relates to displaying students’ work, often in forums. It should be noted that this classifica- 
tion is pragmatic, since it is linked to web functionality rather than student activity. It is, 
therefore, different from student-focused classifications; for example Laurillard’s (2002) 
classifications of media types (attention, inquiry, discussion, practice and production) and 
Conole’s (2007) taxonomy of student activity (assimilative, information seeking and 
handling, modelling, discussion, experiential and productive). 

Our match between these four web functions and the 1 7 web-assisted functions is shown 
in Table 1. Our 63 websites involved all the 17 web-assisted functions (Column 2), and we 
had 115 instances of these in action (Column 5). 

Table 2 shows the diversity of the evaluation instruments used. These favoured self- 
reporting strategies such as surveys and focus-group interviews. As mentioned in McNaught 
and Lam (2005b), we had difficulty persuading teachers that data that directly address 
students’ performance are desirable. Hong Kong has a culture that equates evaluation to 
using student satisfaction questionnaires. 

Three main strategies were used to compare and contrast the lessons learnt in our 63 
cases: 

( 1 ) meta-analysis of the perception of usefulness across all cases; 

(2) meta-analysis of recorded benefits and challenges across all cases; and 

(3) selecting cases that have employed similar strategies and doing a meta-analysis of 
smaller groups of cases. 


Perception of usefulness meta-analysis 

We used an iterative approach to rating each web-assisted function in the context of the 
course in which it was used. In general, the ratings were allocated to each function according 
to the following three criteria: 

(1) Amount of use : The amount of usage recorded concerning that particular function; 
for example, the number of postings in forums, the number of attempts of quizzes, 
or the frequency of access to a particular resource. This information usually came 
from site logs and student surveys. 


66 P. Lam et al. 


<D 


o 


> 

5 

<U> 

£ 

£ 


<D 

-C 


V5 

(D bD 

> P 


3 Oh 

3 g 

O P 

l- 1 -' bh 


t» c/3 

p 

3 a 
c/3 £ 


03 

c 

<D 


U 


■o 

5 


<D 

£ 

p 

C/3 


03 

c 

<D 


c3 

H 


t: 

c3 


P 

O 

o 


(N (N -- Ti h O h 

Tf 


O co O 


(N (N 't O OO 


o VO 

04 


04 

04 


vo co ir> -sf 


€ 


of O- O co 

VO 


o 

03 

d 


O 04 O 04 O ID 


O O O 00 


CO T-f Of ^ uo 

O £> 2 ° o ° 

CD O qd U y <D 

P too S p 5 p 

S p ^ p »g p 

? ^ 7 4 ? 

© g S o S o "d 

o O rv n o, o 

0> 3 (D (□ CD © 

tin h-s GO [JUi C/3 U* H 


03 

d 


P> 

o 

e 

o 

o 


03 

P 


ALT-J, Research in Learning Technology 67 


(2) Quality. The quality of materials; for example, the thread length and the quality 
of the ideas in the forum postings (see Lam and McNaught 2006b for examples); 
the coverage of quizzes, their cognitive demand and amount of feedback; the 
course coverage of the notes and learning resources; and the use of media in the 
learning resources. Judgements of quality involved two reviewers individually 
rating resources or analysing opinions using pre-agreed criteria and negotiating 
differences. 

(3) Perception of users: Opinions were collected from either teachers or students 
(informants) concerning the usefulness of the functions in supporting learning, 
either via a survey question or comments made in interviews. Sometimes, 
informants were asked to rank the functions on a site according to their usefulness 
or when they indicated that they wanted particular functions to stay when the course 
is run again. 

After a significant amount of discussion, one score was assigned to each web-assisted 
function in each case. We used a 1-5 scale, with 1 being ‘not useful’ and 5 being ‘very 
useful’. The interaction between the e3L team was essential here as we were called on by 
our colleagues to justify our decisions for any given case. The judgement involved two 
reviewers reviewing the relevant evidence. There was also, of course, checking between the 
ratings given to a function across a variety of different course contexts. In this level of 
analysis, the perceived usefulness of all the 63 evaluated cases was compared and 
contrasted. 


Meta-analysis of learning benefits and challenges 

The second meta-analysis was carried out to identify the general strengths and weaknesses 
of the eLearning strategies, concentrating on the qualitative data. Comments concerning 
benefits and challenges were usually collected through open-ended questions in surveys, 
focus-group meetings or teachers’ reflections. In a few cases, students’ performance in 
assignments or examinations was also considered. 

The meta-analysis method involved the research team going over the evaluation reports 
for each project and extracting the benefits and challenges mentioned in them. 


Meta-analyses of cases with similar themes 

The meta-analysis of selected cases permitted a closer look at the eLearning strategies and 
the factors essential to their success. Through concentrating on the individual eLearning 
strategies in detail, the analyses were able to provide insights and suggestions concerning 
their use and usefulness in more detail than the overall meta-analyses outlined above. 


Findings 

Perception of usefulness: basic content is favoured 

Figure 1 shows a ranking of the measures of perceived usefulness for the 17 web-assisted 
functions. This is also shown in Table 3 to compare the ranking with our dual classification 
of the functions. The data indicate that the most appreciated features concentrate on the 
provision of content; even provision of simple, static content is more highly valued than 
interactive strategies (four out of the top 10 belong to Tow int. ’). This is in special contrast 


68 P. Lam et al. 



Figure 1. Ranking of perceived usefulness of the various web-assisted functions. 


with those that require human to human communication (‘info int.’ and ‘public’). The only 
exception may be the use of the ‘creation and exhibition of multimedia projects’ strategy, 
ranked third. This highly creative and student-centred activity made it very successful in 
both of the instances studied. 

It is useful to look at the four ‘top’ ranked functions: learning tools such as glossaries, 
notes and PowerPoints, assessment tasks associated with grades, and creation and exhibition 
of multimedia projects. These all have rankings of 3.5 or higher on a five-point scale. 

The highest ranked function is the provision of learning tools such as glossaries. One 
explanation is that, in Hong Kong, English is classified as a foreign language, and the 
standard of English is a matter of concern. It is not surprising that online dictionaries and 
glossaries of technical terms, especially with audio pronunciations or visual material, are 
highly valued. However, as universities in most western countries now cater for a wide 
international student mix, this finding may well be of broader interest. 

The second ranking goes to notes and PowerPoint resources. Requiring low interactiv- 
ity, and often highly didactic, these tools are, however, often treasured by teachers. It 
would seem that, although researchers advocate uses of the web that are interactive and 
seek to develop students’ critical thinking, problem solving, information literacy, 
communication, etc., this will not be valued until the overall curriculum patterns in many 
universities change. Content-dominated curricula and a focus on end-of-course formal 
examinations with predictable patterns year to year are well supported by ‘notes on the 
web’. Students, who understandably wish to succeed in the current system, wisely value 
these resources. 

In the same vein, graded assessment tasks are valued. Without grades, assessment is less 
valued; for example, exhibition of assignments and project/assignment archives get the 
lowest rankings. However, positively, both instances in this group were good quiz 
exemplars, providing useful feedback to students. What is important to Hong Kong students 
is that they get marks for the work they do. The concept of working at quizzes in order to 
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Table 3. Ranking of perceived usefulness of the various web-assisted functions. 


Rank 

Interaction* 

Web-assisted function 

Support, Content, 
Feedback, Discussion 
S C F D 

Average perceived 
usefulness (1-5 scale) 

1 

Low int. 

Learning tools such as 
glossaries 

O 

• 

o 

o 

4.0 

2 

Low int. 

Notes/PowerPoints 

o 

• 

o 

o 

3.8 

3 

Public 

Creation and exhibition of 
multimedia projects 

o 

• 

o 

• 

3.5 

3 

Ss & ‘stuff 

Grade-giving assessments 

o 

o 

• 

o 

3.5 

5 

Low int. 

Required readings 

o 

• 

o 

o 

3.3 

6 

Ss & ‘stuff 

Multimedia materials 

o 

• 

o 

o 

3.1 

7 

Ss & ‘stuff 

Self-assessment 

o 

o 

• 

o 

3.0 

7 

Low int. 

Past papers 

o 

• 

o 

o 

3.0 

7 

Ss & ‘stuff 

Learning resources/links 

o 

• 

o 

o 

3.0 

7 

Public 

Inter-cultural discussions 
(across countries) 

o 

o 

o 

• 

3.0 

11 

Public 

Discussion 

o 

o 

o 

• 

2.7 

12 

Low int. 

Course background 

• 

o 

o 

o 

2.6 

13 

Ss & ‘stuff 

Games 

o 

• 

• 

o 

2.5 

14 

Info int. 

Teacher-student 

communication 

o 

o 

o 

• 

2.3 

15 

Info int. 

Assignment submission 

o 

o 

o 

• 

1.5 

16 

Public 

Exhibition of assignments 

o 

• 

o 

• 

1.3 

17 

Low int. 

Project/assignment archives 

o 

• 

o 

o 

1.0 


*The four types of interactivities: 

Low int. : Low interactivity 

Ss & ‘stuff’: Students interacting with materials and tasks 
Info int.: Interactions about specific information 
Public: More public interaction space 


perform well in other assessments is not popular at all. It is a question of maximising the 
value from all time spent studying. Again, language may be a contributing factor: in many 
Hong Kong courses, the written materials are in English even when classes are in a hybrid 
English and Chinese mode. Students’ reading speeds in a second language restrict the 
amount they wish to read, making them more strategic. Again, however, language 
challenges now exist in virtually all universities, and so this may have wider relevance. 

The fourth highly ranked function is the creation and exhibition of multimedia 
projects. There were two instances of this in our case pool, one in public health and one in 
English. It is heartening that this function was seen to be effective. So, despite the 
comments made above about the overall didactic nature of higher education in Hong 
Kong, if course expectations are clear and oriented towards creative work, students will 
react accordingly. 


Learning benefits: basic content is not favoured 

Despite content-based eLearning strategies being very highly rated for perceived useful- 
ness, the analysis on the level of reported benefits found that communication-rich strategies 
have great potential. Table 4 summarises the learning benefits mentioned in the evaluation 
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reports in the 63 cases. These remarks were further grouped under the four main types of 
eLearning strategies: low interactivity, students and ‘stuff, information interactivity, and 
public (the first column of the table). The numbers in the cells indicate the number of times 
a benefit was recorded for a strategy in our cases. 

There were many comments that the Tow interactivity’ content was able to improve 
learning, ease downloading content, enhance problem-solving and analytical skills through 
the readings, and even change some of the students’ learning habits so that they are more 
ready to review and prepare for lessons. However, comments on these benefits were not as 
strongly or widely voiced as on more interactive types of content and communication 
strategies. 

The interactive ‘Students and “stuff” type of strategies, for example, were considered 
to improve learning through revision and practice. With these strategies, the learning envi- 
ronment became more attractive and learning activities more varied. Multimedia materials 
in particular were considered to be excellent tools for students to view and review operational 
procedures. Self-directed learning was also enhanced; students remarked that they were 
motivated to leam by the new media. 

The benefits of the teacher and student communication in the ‘information interactivity’ 
strategies were also impressive. Students mentioned better learning, increased interest in 
learning, and better communication with teachers. The benefits were not as strong, perhaps 
because there were fewer successfully implemented cases of this type in our study. 

The benefits of the ‘public’ strategies were clear. Students leamt through seeing different 
people’s perspectives, and some through exchanging ideas with people in another culture. 
The learning environment was considered to be very engaging. Many types of learning skills 
were related to this by students: e.g. language skills (students had to express themselves 
clearly in communication), critical skills, argumentative skills and analytical skills. Lastly, 
attitudinal changes such as being more willing to cooperate were noted. 


Challenges: basic content is safe 

The analysis shows great differences among our eLearning strategy types. Table 5 illustrates 
the challenges mentioned in our cases, grouped under the four main eLearning strategies. 
The numbers in the cells are the frequencies of the challenges. 

Interactive content and communication strategies were considered to have more chal- 
lenges than the simple non-interactive strategies. While the Tow interactivity’ strategies 
might fail technically (e.g. when downloading was problematic or slow) or in design (e.g. 
assigning too much reading for the students), in general the strategies were safe and 
successful. 

The interactive ‘Students and “stuff” strategies, however, were demanding in all of the 
four areas identified. For example, in terms of perception challenges, students could treat 
the additional learning materials as unimportant and not pay attention to them. Technically 
speaking, more complicated materials led to problems with correct display on students’ 
computers, and there were database problems and data loss. The importance of a good inter- 
face for complicated tasks was also prominent. As for the learning content, more complaints 
were received about quality and quantity, since interactive materials are generally more 
difficult to create, resulting in fewer being developed and some not reaching high quality 
standards. Remember that seven (all ‘Students and “stuff” types) out of our original 70 
cases failed to complete the evaluation phase. Lastly, there were concerns about out-of-class 
assessment; some students did not find online quizzes fair, particularly if they were used as 
part of the course grades. 


Table 4. Benefits across the 63 cases. 
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- Visual attractiveness 

- Flexible learning 

- Variety of learning tasks 

- More communication 

- Engagement 





Table 4. ( Continued ). 
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- Technology as learning tool 

- Increased interest 

- Willingness to share ideas 

- Discussion as learning tool 
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Table 5. Challenges across the 63 cases. 
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- Inconvenience 

- Problem in displaying content correctly 

- Database problem 

- Poor interface 
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‘Information interactivity’ approaches could be difficult too. These were demanding in 
terms of teachers’ participation, responsiveness to questions, and the quality of feedback to 
students. 

‘Public’ eLearning strategies were also difficult. Students lacked motivation to partici- 
pate, resisted because they prefer face-to-face discussions, found some communication tools 
not to be user-friendly, and perceived a lack of teacher participation, especially when clar- 
ifications were needed. Tasks need clear and adequate guidelines to lead students through 
the activities. Even with a very good design, cooperative group work is usually time- 
consuming. 

Thus ‘benefits and challenges’ analysis indicates that more interactive strategies are also 
considered more challenging to teachers and learners. However, they have the potential to 
deliver more learning benefits than the simpler, non-interactive strategies. 


Lessons learnt from groups of similar cases 

The third type of meta-analysis focuses on studies of cases using similar strategies. Four 
such analyses have been done, focusing on the challenging, interactive strategies. This 
allows the formulation of a list of the factors influencing the success of specific strategies 
across multiple contexts. 

Lam and McNaught (2006a) examined three media-rich cases (a ‘Students and “stuff” 
strategy). The paper reports on the design and evaluation of the online courses and explores 
whether media-rich experiences can enrich learning at the full range of knowledge types and 
levels of cognitive reasoning outlined in the Revised Bloom’s taxonomy (Anderson and 
Krathwohl 2001). There was qualified support for benefits to student learning. The study 
also highlighted factors that influence success: attention to the quality and design of the 
media, considering student motivation, and focusing on feedback on learning during the 
course. Media and learning design thus are inextricably intertwined. 

McNaught et al. (forthcoming) also studied more complex developments of the ‘Students 
and “stuff” type. It is rare to see failures in eLearning projects reported in the literature. 
The e3L cases that were not completed were examined to see why these were unsuccessful. 
The cases were: two sites with large numbers of high-quality video clips; two sites with flash 
animations that had to be scientifically very accurate; one detailed factory simulation; one 
role-play CD-Rom; and one portfolio system. Many challenges were uncovered. Each of the 
different stages (planning and development, implementation, dissemination, and evaluation) 
is critical and we analysed the challenges in each of these stages. Some challenges related 
to human factors - personal (e.g., workload) and interpersonal factors (mostly communica- 
tion issues between teachers and development teams). These challenges can be compounded 
by external factors such as technological infrastructure, university policy and support, and 
the teaching and learning culture as a whole. One can view the development process as an 
intersection of a large number of factors that need to intersect in a mutual comfort zone. 
Sadly, this synergy does not always occur. 

The third study addressed ‘public’ eLearning strategies, involving 13 reasonably active 
educational online forums (McNaught, Cheng, and Lam 2006). Data included student and 
teacher surveys, student focus groups, analysis of forum postings and examination of forum 
log data. Structured forums with a defined purpose generally had a higher quantity and 
quality of postings than free forums; student-centred fora (involving more student-student 
communication) were also of higher quality than discussions where communication was 
mostly from teacher to student. Further, analysing evaluation feedback from students and 
teachers identified three key factors that tend to affect forum success - ease of use, clear 
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facilitation and motivation to engage. The centrality of the role of the teacher was 
confirmed. 

Lastly, Lam and McNaught (2006b) examined web-enabled communication in six cases 
involving peer- and group-assessment activities. These cases involved three web strategies 
- e-resources (using the web to deliver materials for discussion), e-display (displaying work 
for peers to critique) and e-communication (students discussing online). The data generally 
confirm that web-enabled peer- and group-assessment activities can produce positive 
results. However, these assessment activities need careful planning, as several factors affect 
outcomes. For example, the e-display function is not as popular as e-resources (in line with 
the first meta-analysis discussed above). Here, this was due to technical challenges: students 
will only upload work if it is easy to do so. Regarding e-communication, the importance of 
group dynamics and participation was highlighted. The skills needed to work effectively 
online are not the same as basic IT competency skills. In our cases both teachers and 
students were relative novices in online learning, since even students who have spent 
considerable time online may not be experienced in formal online learning situations. 


Discussion 

Table 6 summarises the main observations from the meta-analyses. Web functions that are 
basic and non-interactive, contrary to expectations, were perceived as more useful strategies 
than interactive ones. However, non-interactive strategies supported improvement in basic 
levels of knowledge, skills and attitudes, whereas other strategies were associated with 
learning at the higher levels. So, why would the pedagogically less beneficial strategies be 
more welcome? 


High-risk, high-return 

One explanation, highlighted in Table 6, concerns the high level of investment and risk 
teachers and students take in carrying out interactive strategies. As noted elsewhere, skills 
are needed in developing (Falconer 2007) and implementing online strategies (Moule 2007). 


Table 6. Combined observations in the three meta-analysis methods. 



Perceived usefulness 
(meta-analysis 1) 

Benefits Challenges 

(meta-analysis 2) (meta-analysis 2) 

Highlighted factors/issues 
(meta-analysis 3) 

Low int. 

Often high 

Basic 

Low 

- 

Ss & ‘stuff 

Ranged between high 
and low 

Can be very high 

High 

• Media quality 

• Learning design 

• Richness of feedback 

• Interaction with human, 
technology, 

environment and culture 

Info int. 

Ranged between high 
and low 

Can be high 

Medium 

— 

Public 

Ranged between high 
and low 

Can be very high 

High 

• Learning design 

• Group dynamics 

• Teacher and student 
participation 

• Support 

• Online learning skills 
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The fact that these strategies are challenging to develop and implement was confirmed 
in our third type of study, which examined more complex ‘Students and “stuff” and 
‘Public’ strategies. Expected learning benefits were broadly confirmed in successful appli- 
cations of these strategies, but various factors affect these outcomes such as the quality of 
the materials (particularly for multimedia), the design of activities, the readiness of teachers 
and students to participate and the support available. 


Using a meta-analytic approach 

The findings reveal that greater understanding can be achieved through comparing and 
contrasting naturalistic studies of eLearning use. Quantitative data may not always be 
directly comparable, as instruments and data vary. Qualitative data, on the other hand, 
yielded categories of description that could be analysed. 

Pragmatic meta-analysis can also support generalisation across cases in terms of over- 
all usage patterns, what works well and what does not (challenges), the strengths and 
weaknesses of a certain innovation and strategy, and factors influencing outcomes. This 
makes use of the many isolated and fragmented naturalistic studies that exist, and may 
help practitioners become better informed about the designs of innovations and eLearning 
strategies. 

There are obvious limitations to this approach. As mentioned above, our naturalistic 
eLearning studies were heavily biased towards perception data rather than data on actual 
perfonnance. Oliver and Conole (2003) called these “proxies for performance” and there is 
a tendency “to measure what is easily measured” (389). Generalisability of meta-analysis of 
this kind depends on both the quality and quantity of the individual studies available. 

It is also interesting to note that pragmatic meta-analysis is not restricted to the study of 
completed or successful cases. As illustrated by our study of the seven incomplete cases, 
less satisfactory experiences can be very revealing. 


Conclusion 

We have suggested three pragmatic strategies for meta-analyses of naturalistic evaluations. 
Naturalistic eLearning studies vary in the evaluation techniques used, but the meta-analyses 
described in this paper can integrate findings and insights from studies with quite different 
designs. 

This study indicates that basic and non-interactive eLearning strategies are often most 
welcome, while perceptions of more beneficial interactive strategies can fluctuate. This 
reflects that interactive strategies require support, and skills in development and implemen- 
tation. It is a high-risk, high-return approach, and it seems that few teachers are currently 
willing to take this risk. However, several factors can interfere with these complex strategies. 
There are lessons for teachers embarking on an eLearning project, staff who support the use 
of technology in universities - both in tenns of technical support and educational design - 
and those responsible for developing institutional policy about the effective use of technol- 
ogy in teaching and learning. 
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